[FEA] Support yyyyMMdd under CORRECTED policy [databricks] by gerashegalov · Pull Request #15820 · NVIDIA/cudf-spark

gerashegalov · 2026-08-28T23:31:49Z

Follow-up issue: #15977

Description

The yyyyMMdd format currently falls back to CPU under the CORRECTED time parser policy unless incompatible date formats are enabled. The customer workload described in #15588 was 14.9% faster when the existing GPU path was forced.

This change:

certifies yyyyMMdd for the fused JNI parser under CORRECTED when spark.rapids.sql.hasExtendedYearValues=false;
keeps yyyyMMdd on CPU under EXCEPTION so Spark can detect CORRECTED/LEGACY disagreements such as 2024101;
uses explicit, direction-specific compatibility allowlists so parsing certification cannot silently certify formatting, or vice versa;
keeps MMyyyy on CPU under EXCEPTION after confirming that input 12024 throws on Spark CPU but returns null in the fused GPU parser;
limits the packed-year parsing guard to CORRECTED, preserving the existing LEGACY incompatible-format opt-in;
falls back for year-bearing formatting when extended years may be present, while retaining the explicit incompatible-format opt-in;
adds table-driven coverage for every EXCEPTION-policy allowlisted format across all supported timestamp/date operators and ANSI settings.

Issue #15977 tracks restoring MMyyyy EXCEPTION acceleration after the fused parser preserves parser-policy disagreements.

Local verification:

mvn -T64 -pl sql-plugin -am -DskipTests verify
mvn -T64 package -pl dist,integration_tests -am -DskipTests -Dbuildver=357
mvn -T64 -f scala2.13/pom.xml package -pl dist,integration_tests -am -DskipTests -Dbuildver=401
Spark 3.5.7 focused compatibility tests: 22 passed
Spark 3.5.7 EXCEPTION-policy matrix: 140 passed
Spark 4.0.1 focused compatibility tests: 23 passed
Spark 4.0.1 EXCEPTION-policy matrix: 140 passed
Python syntax validation and git diff --check

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov · 2026-08-29T00:41:34Z

build

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov · 2026-08-31T18:29:08Z

build

greptile-apps · 2026-08-31T18:32:00Z

Confidence Score: 5/5

The PR appears safe to merge based on the available follow-up evidence.

Summary

Separates parsing compatibility for CORRECTED and EXCEPTION policies from formatting compatibility.
Adds extended-year safeguards for parsing and formatting paths.
Expands integration coverage for valid and invalid inputs, ANSI behavior, fallback, Spark 4 defaults, and extended years.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Date/time expression with literal format] --> B{Operation direction}
    B -->|Parsing| C{Parser policy}
    B -->|Formatting| D[Formatting compatibility set]
    C -->|CORRECTED| E[CORRECTED compatibility set]
    C -->|EXCEPTION| F[EXCEPTION compatibility set]
    C -->|LEGACY| G[LEGACY compatibility set]
    E --> H{Format compatible and year range allowed?}
    F --> H
    G --> H
    D --> H
    H -->|Yes| I[GPU expression]
    H -->|No| J[CPU fallback]

_{Reviews (3) · Last reviewed commit: "Harden timestamp format compatibility ch..."}

rishic3 · 2026-09-01T20:30:52Z

    "MM-yyyy",
    "MM/dd/yyyy",
    "MM-dd-yyyy",
+    "yyyyMMdd",


This list claims to be for both CORRECTED and EXCEPTION, and then there is a separate list for LEGACY, but I think that is misleading. EXCEPTION is supposed to:

try the CORRECTED parser (returning on success)

if it fails try the LEGACY parser

if the LEGACY parser succeeds throw SparkUpgradeException

Per step 2 that means being supported under EXCEPTION also requires LEGACY to produce valid success/failure semantics. E.g. AI came up with the combination of 2024101 to yyyyMMdd under EXCEPTION, which should fail (Spark LEGACY accepts it) but we succeed (the JNI throws on it in LEGACY). I think we need three compatibility lists: CORRECTED, EXCEPTION, and LEGACY, where atm yyyyMMdd is not EXCEPTION compatible.

Good catch. Fixed in 18bd197 by separating EXCEPTION_COMPATIBLE_FORMATS from CORRECTED_COMPATIBLE_FORMATS and selecting the policy-specific set during both tagging and execution. yyyyMMdd now falls back to CPU under EXCEPTION, so Spark preserves the CORRECTED/LEGACY disagreement behavior. I added coverage that asserts GetTimestamp fallback for a normal yyyyMMdd value and the expected error for 2024101. The focused Spark 3.5.7 GPU run passed.

Looks good for yyyyMMdd. But we should check the other formats to see if anything else slipped by, with improper legacy semantics. I highly suspect many of the other formats are also wrong for EXCEPTION. Perhaps deferred to an issue + TODO comment.

Audited and hardened in a880d83. EXCEPTION_COMPATIBLE_FORMATS is now an explicit allowlist rather than a subtraction from CORRECTED, and each of its 14 formats has a representative CORRECTED/LEGACY disagreement case exercised across five operators and ANSI on/off (140 cases). The matrix passed on Spark 3.5.7 and 4.0.1. The audit confirmed an additional mismatch for MMyyyy with input 12024, so that format now falls back under EXCEPTION. Issue #15977 and the adjacent TODO track restoring it after the fused parser preserves the disagreement.

rishic3 · 2026-09-01T20:42:31Z

    "MM-yyyy",
    "MM/dd/yyyy",
    "MM-dd-yyyy",
+    "yyyyMMdd",


On a separate note this list is also used by date_format, not just parsing. I don't know if we've verified that the reverse direction date_format(timestamp, 'yyyyMMdd') matches Spark.

Verified and covered in 18bd197. yyyyMMdd is now included in the direct date_format parity matrix for both Date and Timestamp inputs, including the runtime-fallback and timezone-rule cases. The focused Spark 3.5.7 GPU run selected 25 yyyyMMdd cases: 24 passed and the Spark-4-only case skipped, with no failures or errors.

For the signed/extended years we are not matching Spark:

SELECT date_format( timestamp_micros(327403411200000000L + id), 'yyyyMMdd') FROM range(1)

CPU: +123450101, GPU: 23450101.

Fixed in a880d83. Formatting now has its own compatibility direction, and any certified format whose converted pattern contains %Y falls back when spark.rapids.sql.hasExtendedYearValues=true, unless the existing incompatible-date-formats opt-in is enabled. I added extended-year fallback coverage for all 12 certified year-bearing formatting patterns plus GPU parity checks for the four patterns without a year. The focused suites passed on Spark 3.5.7 (22 tests) and Spark 4.0.1 (23 tests).

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

…-20260902

gerashegalov · 2026-09-03T14:58:25Z

build

gerashegalov · 2026-09-04T14:58:41Z

build

rishic3 · 2026-09-11T18:00:07Z

+          // The fused parser only accepts an unsigned four-digit year for this packed format.
+          if (expr.left.dataType == DataTypes.StringType && sparkFormat == "yyyyMMdd") {
+            YearParseUtil.tagParseStringAsDate(conf, this)
+          }


Before this change LEGACY + spark.rapids.sql.incompatibleDateFormats.enabled=true accelerated yyyyMMdd without also requiring spark.rapids.sql.hasExtendedYearValues=false. I think this guard is overbroad

Fixed in a880d83. The packed yyyyMMdd extended-year parsing guard now applies only under CORRECTED. LEGACY with spark.rapids.sql.incompatibleDateFormats.enabled=true retains the existing acceleration behavior even when spark.rapids.sql.hasExtendedYearValues=true; a regression test asserts CPU/GPU parity and GPU execution for that configuration on Spark 3.5.7 and 4.0.1.

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Support yyyyMMdd under CORRECTED policy

5e0ccde

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov added feature request New feature or request AI draft labels Aug 28, 2026

gerashegalov self-assigned this Aug 29, 2026

Fix yyyyMMdd legacy integration tests

1e57413

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov requested a review from a team August 31, 2026 18:29

gerashegalov marked this pull request as ready for review August 31, 2026 18:29

gerashegalov changed the title ~~[WIP] [FEA] Support yyyyMMdd under CORRECTED policy [databricks]~~ [FEA] Support yyyyMMdd under CORRECTED policy [databricks] Aug 31, 2026

gerashegalov requested a review from rishic3 September 1, 2026 00:44

rishic3 reviewed Sep 1, 2026

View reviewed changes

gerashegalov added 2 commits September 1, 2026 17:15

Respect EXCEPTION datetime format semantics

18bd197

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Merge remote-tracking branch 'origin/main' into codex/pr15820-upmerge…

d6b7179

…-20260902

gerashegalov requested a review from rishic3 September 4, 2026 20:50

rishic3 reviewed Sep 11, 2026

View reviewed changes

gerashegalov mentioned this pull request Sep 11, 2026

[BUG] Fused MMyyyy parser misses EXCEPTION-policy disagreement #15977

Open

Harden timestamp format compatibility checks

a880d83

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Uh oh!

Conversation

gerashegalov commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

gerashegalov commented Aug 29, 2026

Uh oh!

gerashegalov commented Aug 31, 2026

Uh oh!

greptile-apps Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gerashegalov commented Sep 3, 2026

Uh oh!

gerashegalov commented Sep 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gerashegalov commented Aug 28, 2026 •

edited

Loading

greptile-apps Bot commented Aug 31, 2026 •

edited

Loading